Skip to content

fix!: apply the accepted GeoZarr reader findings from EOPF-Explorer/data-model#349 - #360

Draft
d-v-b wants to merge 48 commits into
EOPF-Explorer:mainfrom
d-v-b:claude/issue-349-fixes
Draft

d-v-b wants to merge 48 commits into
EOPF-Explorer:mainfrom
d-v-b:claude/issue-349-fixes

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖

This fixes the findings accepted from the independent reader's review in #349: #351, #353, #355, #356 and #357 in the S2 output, and #352 across the S2, S1 RTC and OLCI converters. It is stacked on #359 (the plan-driven writer), which is stacked on #330, and merges after both.

Warning

#352 breaks CRS resolution in rioxarray as released (0.23 and 0.24, and its main branch). rioxarray recognizes the proj convention only under the name "proj:", while the proj v0.1 spec and zarr-cm name it "proj". With the spatial_ref grid-mapping array gone, ds.rio.crs is None for this output. The CRS is still declared, as proj:code on each group. A strict xfail test (test_rioxarray_resolves_the_level_crs) records the break and fails once a rioxarray release reads the spec's name.

What changed, by issue
  • spatial:shape on multiscale parent groups #355: the multiscale parent groups state the spatial:shape their spatial:transform describes.
  • Attribute hygiene in S2 output: flag_meanings, short_name, dtype, valid_min/valid_max #351:
    • A flag_meanings list becomes one blank-separated string, the SCL table included.
    • The dtype attribute is dropped.
    • A valid_min / valid_max outside the integer data type's range is dropped.
    • A carried short_name is written per level, as <variable>_r<resolution>m, coarsened levels included.
    • The first three rules live in one helper, normalize_cf_array_attrs, which the shared sanitize_array_attrs also applies.
  • Do not shard arrays that fit in one chunk #357: an array that fits in one inner chunk is written in plain chunks, not as a one-chunk shard. That was 201 arrays in the fixtures.
  • Store the geometry label coordinates as Zarr string #356: fixed-length unicode and bytes labels (angle, band, detector) are stored as the registered string data type. The writer decodes bytes as ASCII.
  • Add proj and spatial conventions to /conditions/meteorology #353:
    • The meteorology groups get proj and spatial attributes derived from their latitude/longitude values. They use node registration, signed steps, unwrapped longitudes, and a bbox within [-180, 180].
    • They lose the source Conventions: "CF-1.7".
    • A grid no transform describes is carried without georeferencing, with a warning, rather than failing the conversion.
  • Remove GeoZarr V0 grid_mapping and spatial_ref from the output #352:
    • No spatial_ref arrays and no grid_mapping attributes are written by the S2 plan, the S1 RTC ingest, the OLCI converter or the shared write path.
    • strip_cf_grid_mapping removes them before writing. When a dataset has no proj convention key, it first carries the grid mapping's CRS over as proj:code.
    • The S1 model no longer requires spatial_ref, but still accepts it, so stores ingested before validate.
    • The docs, the OLCI notebook and CHANGELOG.md (Unreleased, Breaking) mark the grid mapping obsolete.
Design notes and scope
  • The meteorology georeferencing needs coordinate values, not just metadata. The plan reads only metadata, so plan_geozarr takes an optional coordinate_values lookup, and write_geozarr passes its source reader's. Without the lookup, those groups get no georeferencing.
  • Remove GeoZarr V0 grid_mapping and spatial_ref from the output #352 leaves the generic converter alone. create_geozarr_dataset, behind the CLI convert generic path and the CPM generic pipeline, is built around the GeoZarr V0 grid mapping and still writes it. Its contract test cases are marked that way.
  • The validators are unchanged. check_grid_mapping checks only a declared grid mapping, and the minispec validator tolerates one, so older stores still validate.
  • sanitize_array_attrs keeps valid_* on float arrays. Its callers pass is_decoded_float for native floats, whose ranges are in physical units, so only its docstring was corrected.
  • Fixtures were regenerated: the S2 snapshots, the OLCI snapshot and the S1 RTC fixture. The S1 fixture also picks up _FillValue and resampling_method attributes the ingester was already writing; the checked-in copy was stale.
Verification

roborev reviewed every commit, and the agent fixed each valid finding. The low findings on the writer that are still open are listed in #359. On the current head, the agent ran:

Check Result
Full test suite (not network), with the venv on PATH 899 passed, 5 skipped, 3 xfailed (one is the rioxarray break)
CPM writer tests, in an environment with the cpm extra 40 passed
pyright clean
ruff 0.14.3 check and format clean

Before the merge with the split writer branch, the agent also recounted the #349 findings in the planned output of the V2 snapshots and the native V3 fixtures: none were left, except the meteorology groups without coordinate values in the snapshot test. That recount was not repeated after the merge.

Stack

🤖 Generated with Claude Code

d-v-b and others added 30 commits October 28, 2026 12:28
…tic-zarr main

pydantic-zarr now validates every Zarr V3 array spec as a whole metadata
document, judging each extension-point name by the definition that
claims it in a scope. The GeoZarr DataArray sets zarr_scope_strict, so
a typo'd or unknown codec or data type name is refused instead of
passing unjudged, against a GeoZarr scope that extends the default with
zarr-python's provisional fixed_length_utf32 data type (band and angle
coordinates in real stores hold it, and no zarr-metadata scope claims
it yet). The pyz wrappers drop their attributes redeclaration: the
parent field now carries a validated empty-dict default, so narrowed
attribute types that refuse {} stay required without it.

pydantic-zarr comes from git main pending its next release.

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bers override

The pyz ArraySpec type parameter was phantom (both wrappers subclassed
ArraySpec[Any]), so every declared attributes model on the Sentinel
array classes was decorative: Sentinel2DataArray accepted arbitrary
attributes and held plain dicts. The parameter is now passed through,
so attributes validate against the declared model and read back typed.
The GroupSpec wrappers stop binding the members mapping type into the
parent's member-item slot (the old type-var hack and both its ignores):
with pydantic-zarr's covariant parameters, parameterizing the parent at
its widest member type and narrowing the members field is a legal
override. s3_olci spells unconstrained attributes Mapping[str, object]
(object no longer satisfies the enforced bound), and the GeoZarr v3
group classes use AnyGroupSpec, now an ordinary wide type, so member
reads stay checked. pydantic-zarr is bumped to main with covariant
spec parameters (zarr-developers/pydantic-zarr#191).

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
load_sentinel2 reads a Zarr store into the validated Sentinel2Root
input model — keeping the model as the carrier instead of validating
and discarding it — and plan_geozarr is a pure function from that model
to the GeoZarr hierarchy to write, as a Zarr V3 GroupSpec tree: the
metadata plane of the conversion, independent of chunk copying. The
reflectance pyramid (native levels with the product level's finer bands
injected, coarser levels derived, sharded uint16 bands, float64
coordinates, CF grid mappings from the product's own proj:wkt2 via a
CF round-trip, spatial/proj/multiscales convention attributes) is
derived entirely from the input model's metadata, and reproduces the
converter's stored snapshot byte for byte for the product whose tile
the snapshot harness models; for the other products the snapshots carry
the harness's synthetic tile georeferencing, so the tests mask those
values and pin the plan's georeferencing to the product's own band
attributes instead. conditions and quality stay on the DataTree
converter for now.

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…quality

plan_geozarr now plans the whole hierarchy. The aux multiscale groups
(l1c/l2a classification, cloud probabilities) get the same pyramid
treatment as reflectance via AUX_MULTISCALE_LEVELS — native levels
carried, coarser levels derived with attributes verbatim, nearest
resampling when the arrays carry a flag table — and every other
conditions/quality dataset is carried: georeferenced datasets (detector
footprints, quality masks, atmosphere) gain spatial attributes and a
spatial_ref grid mapping derived from their own proj attributes,
geometry and meteorology are translated as-is (string arrays to
fixed_length_utf32, coordinates uncompressed, data arrays sharded), and
the quicklook groups and scalar arrays are dropped, as the converter
drops them. The planner walks the input model's dump, since the input
model types these subtrees loosely. Runtime assert statements are
replaced with raises or restructured away.

The whole planned tree is byte-for-byte equal to the stored converter
snapshot for the product whose tile the snapshot harness models; for
the others the tests mask the harness's synthetic-tile georeferencing
and the generated aux levels' chunk shapes, which the converter derives
from incidental dask chunking (inner chunks of 915, 512 and 1830 for
the same situation across the snapshots) where the plan uses its
uniform rule.

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…re encoding

Coarsening divides whatever dask chunks the source dataset happens to
have, so the stored encoding of a generated level depended on the
incidental chunking of the object it was coarsened from: the snapshots
held inner chunks of 915, 512 and 1830 for like levels. generic
multiscales now rechunks the coarsened dataset to the spatial chunk
target (min(spatial_chunk, dim) per dimension) before deriving the
encoding, matching the native levels' policy and the spec-level plan's
uniform rule. The snapshots are regenerated (the only other change is
the omission of null dimension_names, which current pydantic-zarr no
longer serializes), and the plan tests drop their generated-level
chunk masking: the whole tree is now pinned.

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Picks up the unreleased extra-fields support (consolidated root documents
now validate, carried in extra_fields and refused by to_zarr), the typed
to_metadata_json / to_stored_metadata_json exports, the always-complete
GroupSpec hierarchy, and whole-document validation for Zarr V2 arrays.
Whole-repo pyright and the full non-network suite are green against it.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…data and zarr-cm

Every zarr metadata field the Sentinel-2 plan writes now carries its
published type: zarr-metadata's *Object TypedDicts for the extension
points (regular chunk grid, default chunk key encoding, and the bytes,
blosc, zstd, crc32c and sharding_indexed codecs), zarr-cm's convention
types for the zarr-conventions attributes (the CMO declarations replace
the hand-written registry dicts, and layout entries are LayoutObject
with the spatial convention's keys as typed extra items), and dedicated
TypedDicts for the fixed-shape documents the plan builds itself:
GridMappingAttrs (narrowed at runtime from pyproj's CF output via
zarr_metadata.typed_json.check), SpatialGroupAttrs,
MultiscaleGroupAttrs, and FixedLengthUtf32Object in the GeoZarr scope.
Function-scoped imports move to the top of their modules. No runtime
change: the stored snapshots are byte-identical.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds data_api/s2_write.py: write_geozarr realizes the plan with
zarr-python and numpy alone. The planned metadata documents are written
first (pydantic-zarr to_zarr, so every stored document is exactly the
plan), then the chunk data is filled in: native arrays copied from the
input store, injected bands block-resampled from the finest input level
that holds them, generated pyramid levels chained from the level before
them with the converter's arithmetic (nodata-masked rounded block mean,
window-centre subsampling for classifications, window means for the
coordinates), and grid-mapping arrays set to zero. Product-level
publication (root attributes, consolidation, STAC) stays the converter
pipeline's separate final step.

The parity test runs write_geozarr and create_multiscale_from_datatree
on the same seeded input (each product's own proj:transform drives the
coordinates; small arrays and the scene classifications carry patterned
values) and compares the stores file by file: identical file sets,
identical metadata documents, and byte-identical chunks, for all three
fixture products.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Converts the pyz GroupSpec/ArraySpec wrappers from Generic[...] bases
with module-level TypeVars to PEP 695 type parameters (ruff UP046), and
drops an unused import this branch had left in geozarr/v3.py.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…move the DataTree writer

convert_s2_optimized now validates the input store into Sentinel2Root,
plans the output with plan_geozarr, and writes it with write_geozarr —
metadata documents first, chunks after, root consolidation and STAC
unchanged as the final step, fed by a dataset index derived from the
plan. The DataTree-based write path (create_multiscale_from_datatree,
generic_multiscales, stream_write_s2dataset, inject_missing_bands, the
coarsening kernels and write_geo_metadata) is removed, after the
plan-driven writer reproduced its stores byte for byte; s2_multiscale
keeps only the declarative policy tables and the SCL vocabulary.

The write policy is the plan's fixed policy (sharding on, 1024-pixel
inner chunks, blosc-zstd level 3, ESA packed-integer layout): deviating
option values are refused with a message naming them, the layout flags
are gone from the convert-s2-optimized CLI command and ignored on the
auto-detect convert path, and the CPM writer forwards only the options
the S2 pipeline still interprets.

The migration also widens the inputs the plan accepts, which the
DataTree path tolerated and the Zarr V3 (N0512) fixtures exercise: a
tree with no backing store or a non-V2 backing store is materialized to
a scratch V2 store (minus its per-variable zarr storage encoding),
conditions/meteorology is optional in the input model, georeferencing
falls back to the product-level stac_discovery proj declaration when
the band and mask arrays carry none, and numpy S dtypes map to
zarr-python's null_terminated_bytes, now claimed by the GeoZarr scope.

Tests: the writer tests pin plan realization and the resampling
semantics against independently formulated references (the byte parity
against the DataTree writer was proven before its removal); snapshot
regeneration moves to the plan tests; the multiscale tests keep the
encoding and end-to-end structural coverage; the geo-metadata unit
tests go with the code they tested.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…and drop the scratch copy

The converter no longer materializes in-memory or Zarr V3-backed trees to
a scratch Zarr V2 store. New `pyz.datatree` is the DataTree analogue of
`GroupSpec.from_zarr`: `group_spec_from_datatree` extracts a tree's
metadata plane as the Zarr V2 documents xarray's own zarr writer would
store (every variable through `encode_zarr_variable`, CF packing applied,
storage-layout encoding keys dropped), so `Sentinel2Root` validates a
tree's named members and typed attributes exactly as it validates a store.
`load_sentinel2` accepts a DataTree, and `write_geozarr` reads the input
through a `SourceReader` protocol with a zarr-store and a DataTree
implementation, the latter reading values in the same encoded form.

A parity test pins the equivalence: on the seeded Sentinel-2 fixtures,
the tree source writes the same store as the zarr-store source, chunk for
chunk, whether the tree holds the raw packed values or the decoded ones
(which the reader re-encodes as xarray would have on a store write).

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Correctness and robustness:
- `_window_mean` keeps an all-NaN float window NaN instead of writing 0.0,
  matching the removed converter's skipna mean and `_coarsen`'s docstring.
- `_plan_subtree` carries a mixed group's own arrays alongside its planned
  subgroups instead of dropping them silently.
- `plan_reflectance` raises its deliberate ValueErrors (no band arrays;
  missing proj:transform/proj:epsg/proj:wkt2 with no product fallback)
  instead of bare StopIteration/KeyError.

Consistency and single sources of truth:
- Root consolidation consolidates every multiscale root the plan maps to
  None, dropping the converter's dependency on the duplicate
  `_AUX_MULTISCALES` table; the dead `REFLECTANCE_MULTISCALE` is deleted.
- The CPM writer forwards spatial_chunk and compression_level to the S2
  policy check, so an explicit deviation is refused loudly rather than
  silently ignored (enable_sharding stays unforwarded: sharding is fixed on
  and its CPM default of False is indistinguishable from an explicit False).
- The pydantic-zarr git source pins the validated commit (9f2e847) instead
  of floating on main.

Snapshots and docs:
- The plan snapshots are regenerated from the plan, replacing the snapshot
  harness's synthetic-tile georeferencing with each product's own; the
  masking machinery and `_EXACT_PRODUCT` special case are retired, so every
  product's snapshot comparison is now exact.
- The docs drop the removed convert-s2-optimized layout flags, scope
  --scale-offset-codec to the generic rechunker, document the fixed S2
  write policy, and replace the deleted create_multiscale_from_datatree
  reference with the plan/write API.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…to feat/pydantic-zarr-scoped-validation

Resolves the conflicts between this branch's plan-driven S2 pipeline and
upstream's root-consolidation harmonization (EOPF-Explorer#328) and spatial-chunking fix
(EOPF-Explorer#331):

- convert_s2_optimized now feeds the planned dataset index into the
  harmonized utils.updated_root_consolidation, passing the plan's None-mapped
  multiscale roots as additional_consolidate_paths; the converter-local
  simple_root_consolidation, add_multiscale_pyramids_to_stac_metadata and
  write_store_root_bbox are gone, and add_s2_stac_assets (from EOPF-Explorer#328) is
  adapted to take the index and the planned reflectance shape.
- utils.add_multiscale_pyramids_to_stac_metadata accepts a dataset's variable
  names in place of the dataset, which is what the plan-driven converter has.
- convert_s2 and the DataTree-writer code EOPF-Explorer#328/EOPF-Explorer#331 touched stay deleted
  (no remaining callers); max_retries leaves convert_s2_optimized as upstream
  removed it from the call sites.
- tests follow: the consolidation tests run against updated_root_consolidation
  with explicit None roots, upstream's revived test_s2_multiscale_geo_metadata
  (now against conversion.utils) is kept, and the DataTree-writer tests stay
  deleted.
- uv.lock re-locked: upstream's bumps plus this branch's pinned pydantic-zarr
  commit; the regenerated plan snapshots are byte-identical to the merged
  files, so both pipelines had converged on the same output.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Pin pydantic-zarr to main at b0efedb, which requires zarr-metadata
  0.8. zarr-metadata is now declared as a direct dependency, since the
  package imports it.
- The GeoZarr scope is zarr-metadata's ZARR_PYTHON_CONTEXT. It claims
  `fixed_length_utf32` and `null_terminated_bytes` with their full
  definitions, which the hand-built scope only approximated, and
  `ZarrV3Context.update` no longer exists. s2_plan types those data
  types with zarr-metadata's `FixedLengthUtf32` and
  `NullTerminatedBytes`.
- The coordinate and grid-mapping builders use `create_default`. It
  supplies the one-chunk grid, the `/` chunk key encoding, the derived
  fill value and the little-endian `bytes` codec those builders spelled
  out by hand.
- Tests copy a spec with the validated `replace`, not
  `model_copy(update=...)`.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Require pydantic-zarr>=0.11.0, sourced from the v0.11.0 tag until the
  release reaches PyPI, instead of a commit on main.
- s2_plan converts a V2 dtype to its Zarr V3 form with
  `parse_dtype_v3(..., scope=GEOZARR_SCOPE)` instead of its own mapping.
  0.11 reads numpy's fixed-length unicode and bytes types as zarr-python
  writes them. The local mapping turned a datetime dtype into an
  unclaimed name such as "datetime64[ns]"; `parse_dtype_v3` gives the
  `numpy.datetime64` form.
- 0.11 accepts zarr-metadata's typed codec objects on `create_default`,
  so the builders that pass them type-check.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A carried V2 `<M8[ns]` array is planned as `numpy.datetime64` with its
unit, the form zarr-python writes and the GeoZarr scope claims. The local
mapping that `parse_dtype_v3` replaced gave the unclaimed name
"datetime64[ns]".

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The five multiscale parent groups carried the finest level's
`spatial:transform` without a `spatial:shape`, so a reader could not tell
what grid the transform described. Both multiscale planners now write
the finest level's shape with its transform. A test checks that every
planned group with a transform also has its shape, and the snapshots are
regenerated (EOPF-Explorer#355).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A `flag_meanings` list is joined into the one blank-separated string CF
  defines. The SCL flag table is written as a string too.
- The `dtype` attribute, which repeats the array's data type, is dropped.
- A `valid_min` / `valid_max` outside the range of the integer data type
  is dropped. The scl `uint8` array's `valid_max: 65535` was one.
- A carried `short_name` is written for the level the array is in, as
  `<variable>_r<resolution>m`. Coarsened levels copied the source level's
  name, and the source styles were mixed.

The first three rules live in one helper, `normalize_cf_array_attrs`,
which the S2 plan and the shared `sanitize_array_attrs` both apply.
`sanitize_array_attrs` now also drops `valid_*` for decoded floats, as
its docstring said. Tests cover the helper's cases, the snapshots'
attributes, and a planted array on native and coarsened levels. The
snapshots are regenerated (EOPF-Explorer#351).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every band and carried data array got a shard, even when the whole array
fit in one inner chunk. Those shards held that one chunk, so a reader made
an index request for nothing. 201 such arrays appeared across the three
snapshots, for example the 9 x 9 meteorology and 23 x 23 angle arrays.
Such an array is now written in plain chunks, with one chunk covering it,
under the same bytes + blosc codecs. Arrays that span several inner
chunks keep their shards, including a shard larger than the array. A
test checks every planned array, and the snapshots are regenerated
(EOPF-Explorer#357).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's findings on a2b8247. That commit made
`sanitize_array_attrs(..., is_decoded_float=True)` drop `valid_min` /
`valid_max`, as its docstring had claimed. But callers pass the flag for
every float array: `create_uniform_encoding` and the generic converter
do, for native floats whose range is in physical units. The packed-integer
output passes it too, and its range is in packed units. So correct
ranges were lost. The removal is reverted, and the docstring now states
what the code does: the decode path that unpacks integers drops `valid_*`
itself. A test pins that a float array keeps an in-range `valid_*`.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g type

The geometry label coordinates (`angle`, `band`, `detector`) carried
their source data types: `fixed_length_utf32` and `null_terminated_bytes`.
The second is in no registry, and a strict Zarr V3 reader must refuse an
unknown data type. A carried unicode or bytes array is now planned as
the registered `string` data type. `create_default` gives it the
`vlen-utf8` codec, the empty fill and one chunk. The writer decodes
bytes labels as ASCII rather than leave the conversion to zarr. A test
checks that no planned array uses either unregistered type, and the
snapshots are regenerated (EOPF-Explorer#356).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CAMS and ECMWF groups under `conditions/meteorology` held arrays on
latitude/longitude dimensions with no CRS. They also carried the source
`Conventions: "CF-1.7"`, which contradicts the output's no-CF-claim
decision.

- A carried group on a regular geographic grid gets `proj` and `spatial`
  attributes derived from its coordinate values. Such a group has 1-D
  `latitude` and `longitude` coordinates and no `proj:*` attributes. The
  attributes are `proj:code` EPSG:4326, the dimensions, the shape, and
  the transform and bbox. The values are grid points, so the
  registration is `node`, and a decreasing latitude gives a negative
  step. Irregular spacing is refused; spacing within a thousandth of the
  step, as float32 storage gives, counts as regular.
- The plan stays a function of what it is given. `plan_geozarr` takes an
  optional `coordinate_values` lookup, and `write_geozarr` passes its
  source reader's. Without the lookup, these groups get no
  georeferencing.
- The source `Conventions` attribute is no longer carried on any group.

Tests cover the derived attributes for increasing, decreasing and
float32 latitudes, the two refusals, and the plan with and without
values. The writer test seeds regular latitude/longitude values, and
the snapshots are regenerated (EOPF-Explorer#353).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on c8c0fbd. The V2 fixtures hold only
unicode labels, so nothing reached the bytes path of EOPF-Explorer#356. A plan test
plants a `|S3` label array and checks that it is planned as `string`
with `vlen-utf8`. A writer test fills an `S3` label array into a string
array and reads back the decoded labels.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pyright refused `np.iinfo` on an unnarrowed data type in the valid-range
guardrail. It also refused `zarr.storage` attribute access and an
untyped array read in the bytes-label write test.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's findings on bbd7637.

- A latitude/longitude group whose values `_lat_lon_attributes` refuses
  (irregular spacing, NaN fill, a single point) stopped the whole
  conversion. Such a group is now carried without georeferencing, as
  before EOPF-Explorer#353, and a warning is logged.
- Longitudes are unwrapped before the spacing check, so a grid crossing
  the antimeridian or 0/360 keeps a regular step and gets its transform.
- `_lat_lon_attributes` returns a `LatLonGroupAttrs` TypedDict, like its
  sibling builders.
- The write test asserts the derived `proj:code` and transform on the
  written meteorology groups, which shows the source reader feeds the
  plan. Plan tests cover an irregular grid, a NaN grid and an
  antimeridian grid.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 3ac4926. Longitudes are unwrapped for the
spacing check and the transform. The bbox took the unwrapped extremes,
so a grid crossing the antimeridian got an east longitude past 180. The
bbox longitudes are now moved back into [-180, 180], so the crossing
reads as west > east, as STAC and GeoJSON spell it.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…referencing

BREAKING CHANGE: no `spatial_ref` grid-mapping arrays (with `crs_wkt` /
`spatial_ref` attributes) and no `grid_mapping` attributes are written
by the S2 plan, the S1 RTC ingest, the OLCI converter or the shared
write path. The `proj` and `spatial` conventions are the only
georeferencing. An array opened on its own has no CRS; read `proj:code`
from its group.

This breaks CRS resolution in rioxarray as released (0.23, 0.24 and its
main branch). rioxarray recognizes the proj convention only under the
name "proj:", while the proj v0.1 spec and zarr-cm 0.5 name it "proj".
With the `spatial_ref` array gone, `ds.rio.crs` is None for this output.
A strict xfail test records the break and fails once a rioxarray release
reads the spec's name (EOPF-Explorer#352).

- S2 plan: no `spatial_ref` members, no `grid_mapping` on groups or
  arrays. `GridMappingAttrs` and the WKT plumbing are gone, so the
  product-level fallback no longer requires `proj:wkt2`. The writer and
  converter drop their `spatial_ref` cases.
- Shared path: `strip_cf_grid_mapping` removes the grid-mapping variables
  and every `grid_mapping` attribute and encoding. `stream_write_dataset`
  applies it, and drops the encoding of the variables it removes.
  `write_dataset_geo_metadata` writes only the convention attributes,
  and now takes a CRS given as a string.
- OLCI: the reprojection keeps rioxarray's CRS in memory for the
  computations downstream. The converter strips the grid mapping before
  each write.
- S1: the ingest no longer adds the grid mapping, and the model no
  longer requires `spatial_ref`. It still accepts it, so stores ingested
  before validate.
- The generic converter (`create_geozarr_dataset`) is built around the
  grid mapping and is left as it is. The output-contract test names it.
- Validators: `check_grid_mapping` checks only a declared grid mapping,
  and the minispec validator tolerates one. Both stay, so older stores
  still validate.
- Docs, the OLCI notebook and the changelog mark the grid mapping
  obsolete. The S2, OLCI and S1 fixtures are regenerated; the S1
  fixture also picks up attributes the ingester already wrote.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 85366f4. Moving a longitude by one turn
left a value more than one turn out of range past 180. That happens when
a wide 0-360 grid starts above 180 and wraps past 0. Longitudes outside
[-180, 180] are now normalized modulo 360. In-range values, the
antimeridian's 180 included, are kept as written. A test covers a wide
0-360 grid.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's findings on 3c881ac.

- `stream_write_dataset` stripped the CF grid mapping even when it added
  no `proj` attributes. That is the case for the generic rechunker, so an
  input whose CRS lived only in the grid mapping was written with no CRS.
  `strip_cf_grid_mapping` now carries the CRS over as the `proj`
  convention attributes when the dataset declares no `proj:*` attribute.
  It drops a `spatial_ref` variable only when that variable is a grid
  mapping.
- The CRS parameters of `write_dataset_geo_metadata` and
  `stream_write_dataset` take a string or EPSG code, matching what the
  code accepts.
- The changelog and architecture docs say that the generic converter
  still writes the grid mapping, and that rioxarray does not yet read the
  proj convention under its spec name.

Tests cover the strip helper (CRS carried over, a declared `proj:code`
kept, an unrelated `spatial_ref` kept) and a stream write of a
grid-mapped input.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d-v-b and others added 18 commits November 13, 2026 21:01
…ping's CRS

Addresses roborev's findings on 2d8e58a.

- The carried `proj` attributes replaced the dataset's `zarr_conventions`,
  dropping a declared spatial or multiscales convention. The proj entry
  is now appended to the declared ones, unless already declared.
- The carry-over is skipped only when the dataset declares a proj
  convention key (`proj:code`, `proj:wkt2`, `proj:projjson`), the rule
  the validator uses. A legacy `proj:epsg` no longer counts.

Test cases cover a legacy `proj:epsg` and a declared spatial convention.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 997b123: the docstring still described the
old any-`proj:*` rule.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pydantic-zarr 0.11.0 is published, so the git-tag source override is
removed and the lock resolves `pydantic-zarr>=0.11.0` from the registry.
The resolved version and code are unchanged.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ixes

Assisted-by: ClaudeCode:claude-opus-5-5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…S2 changes

main changed the DataTree-based S2 write path, which this branch
replaces with the model-based plan. Its changes are ported to the plan
rather than taken as written:

- Atmosphere (EOPF-Explorer#343): `quality/atmosphere` is a multiscale pyramid,
  10-720 m, in the plan's `AUX_MULTISCALE_LEVELS` as in main's
  `_AUX_MULTISCALES`. Only L2A products hold the group, so presence
  gives main's L2A-only rule. `aot` / `wvp` resample as reflectance, per
  main's `determine_variable_type`.
- Geometry (EOPF-Explorer#347, EOPF-Explorer#346): `conditions/geometry` declares the tile CRS with
  node registration. Its shape, transform and bbox come from the grid's
  x/y values: `plan_geozarr` takes an optional `coordinate_values`
  lookup, and `write_geozarr` passes its source reader's. The
  (y, x) angle arrays reference the grid mapping. A grid no transform
  describes is carried without georeferencing, with a warning.
  Resolution levels under `conditions/mask` and `quality` whose arrays
  carry no proj attributes, as in native Zarr V3 products, declare the
  tile CRS with pixel registration from the tile origin.
- Band statistics (EOPF-Explorer#333, EOPF-Explorer#344): computed by the shared root
  consolidation, which the plan path already calls. The STAC asset
  function takes either datasets or the plan's variable names, and keeps
  main's href-based duplicate check and key-collision warning.

Conflicts in s2_multiscale.py and the old-path tests take this branch's
side, since that code was removed. main's new statistics assertion is
kept, and its atmosphere and geometry tests are rewritten against the
plan. The snapshots are regenerated from the plan: the atmosphere and
detector-footprint groups match main's in structure and attribute keys.
The geometry group's value-derived attributes are tested with
coordinate values, in the plan and through the writer. The lock is
updated for the 0.13.1 version bump.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 47265cc. The merge left two parsers of
level names (`r60m`): a regex in s2_plan and string slicing in s2_write.
s2_plan's `level_meters` is now the only one. The writer's
`_level_meters` and `_is_level_name` call it, and `_level_meters`
raises for a non-level name.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR 330 merged EOPF-Explorer main and ported its S2 changes into the plan:
the atmosphere pyramid, the geometry and detector-footprint
georeferencing, and the band statistics. They are carried into this
branch's EOPF-Explorer#351, EOPF-Explorer#352 and EOPF-Explorer#353 work:

- s2_plan: this branch's version, with the port applied on top. The
  geometry grid and the footprint levels get no `spatial_ref` and no
  `grid_mapping` (EOPF-Explorer#352). The tile georeferencing is (origin, EPSG), with
  no WKT. The meteorology (EOPF-Explorer#353) and geometry grids share one
  regular-spacing helper (`_regular_axis`) and one refusal path, which
  carries a grid no transform describes without georeferencing, with a
  warning. `level_meters` is the one parser of level names, for the plan
  and the writer.
- conversion/utils: `strip_cf_grid_mapping` and main's node-transform
  helpers both kept. `write_dataset_geo_metadata` and
  `stream_write_dataset` take main's `registration` and this branch's
  wider CRS type, and write no grid mapping.
- CHANGELOG: this branch's Unreleased section above main's 0.13.x
  releases.
- Tests: both sides kept. PR 330's geometry test now expects no grid
  mapping. Snapshots regenerated.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 47481e8. The `level_meters` keyword
parameter of `_carried_array_attributes` and `_plan_carried_array`
shadowed the module-level `level_meters` once the merge made it public.
The parameter is now `meters`, as in `_with_level_short_name`.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The security check's pip-audit blocked on tornado 6.6
(CVE-2026-126347, fixed in 6.6.1), a transitive dependency through
dask's distributed scheduler. Only tornado moves in the lock.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The writer masked a coarsened integer array's planned Zarr fill value,
which is always 0, out of the block mean. Arrays that declare no nodata
lost their zeros from the mean: the cloud and snow probabilities, where
0 is a valid value, and the atmosphere arrays in products that declare
none. main averages those whole, so 14 coarsened arrays of an L2A
product, and their STAC statistics, differed from main.

The source readers now report the nodata an input array declares, in
main's order: the CF `_FillValue` (a Zarr V2 array's `fill_value`,
xarray's encoding, or the attribute), then the EOPF `fill_value`
attribute. The writer carries that value down each pyramid chain and
skips it alone. An array that declares none is averaged whole.

Tests: both readers over each declaration, and generated probability
and atmosphere levels equal to the plain block mean on seeded values
that hold zeros.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
PR 330 now skips only the nodata the input declares when it coarsens,
so probability and atmosphere arrays that declare none are averaged
whole. The writer's loop takes this branch's side, without the
`spatial_ref` case (EOPF-Explorer#352). The tests keep both sides, and the bytes-label
test's source stub gains the readers' new `nodata` method.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e plan unwired

Restore the Sentinel-2 converter, the DataTree pyramid writer, the CLI, the
CPM writer, their docs, tests and output snapshots to their state on main,
and remove the plan-driven writer (`data_api/s2_write.py`) with its tests.

This branch now only adds the pydantic models: the pydantic-zarr 0.11 and
zarr-metadata 0.8 move, the GeoZarr scope, the Sentinel-2 input model and
the output plan (`data_api/s2_plan.py`). `convert_s2_optimized` writes what
it writes on main. The plan-driven writer moves to a follow-up change
stacked on this one.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Replace the plan's snapshot test with a comparison against what
`convert_s2_optimized` writes. Each Sentinel-2 fixture (one L1C, two L2A)
is shrunk tenfold, given data and converted; every group and array document
stored below the root must equal the planned one. The root's own attributes
are left out, because the converter writes them in its consolidation step.

The comparison found one difference, fixed here: the plan typed every x/y
array as float64, while the converter keeps the input's own coordinate
arrays on native levels (int64 metres in ESA products). The plan now carries
the input's x/y arrays on native levels, data type included, and keeps the
float64 coordinate array for generated levels. A second parametrization
covers a product whose coordinates are float64.

The API reference describes the plan and what it is checked against.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The reflectance plan's docstring said every level holds float64 x/y
coordinates, and `level_meters` named a writer this branch no longer has.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…move the DataTree writer

Bring back the plan-driven Sentinel-2 writer (`data_api/s2_write.py`) that
the previous change set aside, on top of the plan that change checked
against the DataTree writer's output. `convert_s2_optimized` validates the
input into `Sentinel2Root`, plans the output hierarchy, writes the planned
documents and fills the chunk data. The DataTree pyramid writer in
`s2_multiscale.py` is removed.

BREAKING CHANGE: `convert_s2_optimized` has a fixed write policy. A
`spatial_chunk` other than 1024, a `compression_level` other than 3,
`enable_sharding=False` and `scale_offset_codec=True` raise `ValueError`.
The CPM writer's default spatial chunk for the `s2-optimized` pipeline
changes from 256 to 1024.

The output is the same as the DataTree writer's at that policy. The test
that compares the plan with the converter's stored documents passes
unchanged. Converting the three fixture products (shrunk, with data) with
both writers gives identical chunk data and identical documents in every
node; the root's STAC assets differ in one href, where two arrays compete
for the key `b10_r60m`.

The output snapshots are the plan's again, regenerated with the int64 x/y
coordinates that native levels now carry.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…tions before writing

Apply the review findings on the plan-driven writer:

- The CPM writer checks the Sentinel-2 write policy before it prepares the
  target, so a refused option no longer removes an existing product. The
  check is `check_s2_write_policy`, which `convert_s2_optimized` also calls.
- The writer refuses input values whose data type does not cast safely to
  the planned one. zarr casts on assignment, which stored float reflectance
  as 0 and 1 in the planned uint16 band without an error.
- An output location that already holds a different hierarchy raises
  `FileExistsError` saying to remove it, in place of zarr's
  `ContainsGroupError`.
- The quickstart and examples pages no longer pass `spatial_chunk=256`,
  which the policy refuses. The CPM writer's docstring, CLI help and options
  table describe the Sentinel-2 options as fixed.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
PR 330 is now validation-only, and the plan-driven writer lives on
`claude/s2-plan-writer`, which this branch is stacked on. Merging it brings
the conformance test, the int64 x/y coordinates on native levels, and the
writer's new refusals.

Conflict resolution:
- `s2_plan.py`: native levels carry the input's x/y arrays, with no
  `spatial_ref` member (EOPF-Explorer#352).
- `s2_write.py`: values go through the safe-cast check. Fixed-length unicode
  labels pass it into the `string` arrays they are planned as
  (EOPF-Explorer#356).
- The conformance test treats a codec with an empty `configuration` as its
  name-only form, which is how zarr-python stores `vlen-utf8`.
- Snapshots regenerated.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 16, 2026

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant